Image sensing device

ABSTRACT

The image sensing device adapted to compose an appropriate single picture from a plurality of pictures of different exposures obtained from the same subject, wherein the camera operation is controlled using as a reference the signal of one of said plurality of pictures of different exposures.

This application is a continuation, of application Ser. No. 08/386,119,filed Feb. 9, 1995, now U.S. Pat. No. 5,638,118, which is a continuationof Ser. No. 08/260,038, field Jun. 15, 1994, abandoned, which is acontinuation of Ser. No. 08/219,747, filed Mar. 29, 1994, abandoned,which is a continuation of Ser. No. 07/935,046 filed Aug. 25, 1992,abandoned, which is a division of Ser. No. 07/601,014, filed Oct. 19,1990 (U.S. Pat. No. 5,162,914), which is a continuation of Ser. No.07/202,115, filed Jun. 3, 1988 (abandoned).

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates to an image sensing device with a widesubstantial dynamic range.

2. Description of the Related Art

Image sensing devices are being widely used, for example, as the videocamera sections of video tape recorders with integrated cameras or asstill video cameras. A video camera using an image pick-up tube or asolid image sensing element has a relatively narrow dynamic range ascompared with the conventional silver halide photography system, so thatit is subject to generation of white blankings and blackenings (commonnames for portions with excessively high or low luminance) under backlight conditions. In conventional video cameras, adjustment of thequantity of light in such cases is effected by opening the diaphragm twosteps or so, either manually or with a backlighting compensation button.

However, even when such backlighting compensation is appropriatelyconducted, with the main subject being properly exposed, white blankingsmay be generated in the background, leaving the picture with a blankbackground. In other words, the narrowness of the dynamic range in animage sensing device cannot be overcome merely by adjusting the quantityof light to attain a proper exposure for the main subject, as in thecase of conventional devices. To cope with this, an improvement has beenproposed in the conventional image sensing device of the type in which astill image is transduced into an electrical signal by means of a linescanner or the like. According to this improvement, a plurality ofimages of different exposures obtained from the same subject arecombined to compose a single picture. Similar ideas are disclosed, forexample, in U.S. patent application Ser. No. 792,768 and JapaneseLaid-Open Patent Application No. Sho 61-219270.

However, they are directed to still images and are not capable ofproviding a moving picture with a wide dynamic range.

SUMMARY OF THE INVENTION

In view of this problem, it is an object of this invention to provide animage sensing device which has a wide substantial dynamic range andwhich makes it possible to obtain moving pictures.

Another object of this invention is to provide an image sensing devicewhich permits the exposure control and focusing control to be conductedin an appropriate fashion.

To attain the above objects, an embodiment of this invention comprisesimage sensing means, control means adapted to cause a plurality ofpictures of different exposures to be output successively from the imagesensing means and composition means adapted to combine pictures withdifferent exposures from among the ones output from the image sensingmeans to compose a moving picture.

This construction makes it possible to obtain a plurality of images ofdifferent exposures successively, so that by effecting the compositionby means of the composition means in a time less than the time betweensuccessive video frames, composed moving picture of a proper exposurecan be obtained. The speed of the composition operation performed by thecomposition means can be set sufficiently high, so that there is noproblem in this regard.

Another embodiment of this invention consists of an image sensing deviceadapted to compose an appropriate single picture out of a plurality ofpictures of different exposures obtained from the same subject, theimage sensing device being characterized by the fact that the cameraoperation is controlled using a signal indicative of one of theplurality of pictures of different exposures as the reference signal.

The control operations for the camera such as the exposure control andfocus adjustment are thus conducted using the signal indicative of oneof the plurality of pictures of different exposures as the referencesignal, whereby correct camera operation can be assured, without anydestabilization thereof.

A further embodiment of this invention consists of an image sensingdevice adapted to obtain an appropriate picture out of a plurality ofimages of different exposures obtained from the same subject, the imagesensing device being characterized by the fact that the processing ofthe target picture element is effected taking into consideration theinformation on the picture elements around the target picture element.

This device thus provides a dynamic range which substantially includesthe plurality of images of different exposures, so that it can give asatisfactory image both of the main subject and the background. Further,the number of isolated points is reduced by processing the targetpicture element taking into consideration the information on the pictureelements around the target one, whereby a more correct distinction isattained between the main subject and the background.

A still further embodiment of this invention consists of an imagesensing device comprising image sensing means, a control means adaptedto cause video signals indicative of a plurality of pictures ofdifferent exposures obtained from the same subject to be output from theimage sensing means, an image memory capable of storing some of thevideo signals indicative of the plurality of pictures output from theimage sensing means and arithmetic processing means which is adapted,whenever the signal of a target picture element on any picture is judgedto be inappropriate, to select or process signals indicative of aplurality of other pictures corresponding to the target picture elementto generate an output picture element signal.

Thus, by causing video signals indicative of a plurality of pictures ofdifferent exposures to be output from the image sensing means andstoring the video signals temporarily in the image memory, it is madepossible to discriminate the picture elements of an appropriate exposurefrom those of an inappropriate one and to replace the picture elementsof an inappropriate exposure with the picture elements of an appropriateone by replacing the video signal of the former with the correspondingsignal of a picture taken under a satisfactory exposure. Consequently,an appropriate image can be obtained over a wide exposure range, whichmeans that the dynamic range of the image sensing means has beensubstantially enlarged.

A still further embodiment of this invention consists of an imagesensing device equipped with an image sensing means which is adapted tooutput image sensing signals of different charge storage times byswitching for each field and that is characterized by the fact itcomposes a single picture out of successive pictures of two fields.

By thus composing a picture of image sensing signals of different chargestorage times, an image sensing signal of a substantially wide dynamicrange can be obtained.

A still further embodiment of this invention consists of an imagesensing device for obtaining a single picture out of a plurality ofpictures of different exposures derived from the same subject by aswitching operation, the image sensing device being characterized inthat a single picture selection color signal is generated from aplurality of picture selection luminance signals and based on saidsingle picture selection color signal, color signal data are selected.

By thus composing a picture of image sensing signals of different chargestorage times, an image sensing signal of a substantially wide dynamicrange can be obtained.

A still further embodiment of this invention consists of a method ofcontrolling an image sensing device provided with a first image sensingmode in which light storage operations of n seconds duration arerepeated to form a video signal and a second image sensing mode in whichn-second light storage operations and m-second light storage operations(m being less than n) are alternated to form a video signal for a singlepicture, the controlling method being characterized in that a switchingsignal for the first and second image sensing modes and a pictureelement composition control signal for the n-second storage pictures andm-second storage pictures in the second image sensing mode are formedout of the n-second storage pictures.

By thus forming out of the n-second storage pictures in the normal firstimage sensing mode a switching signal for the first and second imagesensing modes and a picture element composition control signal for then-second storage pictures and the m-second storage pictures in thesecond image sensing mode, a first image sensing mode with nodeterioration in the time resolution is selected, and the pictureelement composition in the second image sensing mode is made as freefrom deterioration of time resolution as possible.

Further objects and features of the present invention will be apparentfrom the following description and drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a video tape recorder with an integratedcamera using the first embodiment of this invention.

FIG. 2 is a block diagram of the control circuit 108 of the camerasection of the device shown in FIG. 1.

FIG. 3 is a timing chart of the image sensing element operation.

FIGS. 4(a) and 4(b) are conceptional drawings showing the first andsecond examples, respectively, of the image processing in accordancewith the present invention.

FIGS. 5(a) and 5(b) are block diagrams showing the first and secondexamples, respectively, of the arithmetic circuit 202 in FIG. 1.

FIGS. 6(a), 6(b) and 6(c) are views illustrating the method ofdetermining the threshold for judging white blankings and blackenings.

FIGS. 7(a) to 7(d) are tonal characteristic drawings.

FIG. 8 is a block diagram of the essential part of the second embodimentof this invention.

FIGS. 9(a) to 9(d) are timing charts thereof.

FIG. 10 is a block diagram showing another example of the controlcircuit 108 in FIG. 1.

FIG. 11 is a block diagram showing a variation of the circuit shown inFIG. 2.

FIG. 12 is a block diagram of a video tape recorder with an integratedcamera using the third embodiment of this invention.

FIGS. 13(a) to 13(i) are diagrams showing the signal waveforms of FIG.12.

FIG. 14 is a block diagram of a specific configuration of the selectionflag section of the control circuit 120 shown in FIG. 12.

FIG. 15 is a block diagram showing the entire fourth embodiment of thepresent invention.

FIG. 16 is a block diagram of the processing circuit 732Y shown in FIG.15.

FIGS. 17(a) to 17(g) are timing charts of FIG. 15.

FIG. 18 is a block diagram of the judgment circuit 738 shown in FIG. 15.

FIG. 19 is a view showing the hysteresis characteristics of thereference threshold of the judgment circuit 738.

FIGS. 20 and 21 are views showing examples of area division for the meanvalue calculating circuit 760.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The first embodiment of this invention will now be explained withreference to the drawings. FIG. 1 is a block diagram of an entire videotape recorder with an integrated camera to which the present inventionis applied.

In FIG. 1, a camera section is shown at 100, a processing section at 200and a recording section at 300. In the camera section 100, an incidentlight beam entering through an optical system 101 is restricted inintensity by means of a diaphragm 102 and forms an image on an imagesensing element 103. The image sensing element 103 is composed ofsemiconductor elements such as MOS and CCD. A focus drive circuit 107, adiaphragm drive circuit 106 and an image sensing element drive circuit105 drive the optical system 101, the diaphragm 102 and the imagesensing element 103, respectively. A camera signal processing circuit104 is a well known circuit adapted to perform γ correction, etc., likethe camera signal processing circuit of an ordinary video camera.

The image signal output from the camera section 100 is converted into adigital signal by an A/D converter 201 in the processing section 200 andundergoes a picture element transduction operation, which will bedescribed later, in an arithmetic circuit 202. Then, it is restored toan analog signal by a D/A converter 203 and supplied to the recordingsection 300. At 204 is shown an image memory for the operation in thearithmetic circuit 202, and at 205 an addressing circuit thereof. Theaddressing circuit 205 outputs address control signals for writing andreading at the image memory 204 in accordance with a timing signal froma control circuit 108 in the camera section 100.

In the recording section 300, the analog signal from the D/A converter203 is recorded in a video tape recorder 301 in a well known manner.

Next, the operation of the image sensing element 103 will be explained.FIG. 2 is a block diagram showing the camera section 100 in more detail,and FIG. 3 is a timing chart of the camera section 100, taking the NTSCsignal as an example. A field index (FI) signal is a signal fordistinguishing between odd and even fields composing one frame. TheV_(BLK) signal is a vertical blanking signal; the high (H) period of thesignal corresponds to the effective picture and the low (L) part thereofcorresponds to the vertical blanking period. TPULSE is a signal forcontrolling the charge storage time of the image sensing element 103;for example, in the case of an interline type CCD image sensing element,it is a pulse for reading out the picture element output to the CCD forvertical transmission. During the period between a and b of T_(PULSE),the charge in the CCD for vertical transmission is reversely transmittedto the upper drain, and during the period between b and b or thatbetween b and a, the charge in the CCD for vertical transmission istransmitted to a horizontal shift register at a rate fast enough tosupport television transmission. An iris gate signal is a signal forspecifying whether the storage signal of 1/1000 sec. or that of 1/60sec. is to be selected as the reference signal for the automaticexposure, which is to be described later.

In the example shown here, a 1/1000 sec. storage is performed during thevertical blanking period and the 1/1000 sec. storage signal is outputduring the next effective picture period. Immediately after the 1/1000sec. storage period, a charge storage of substantially 1/60 sec. iseffected to output the 1/60 sec. storage signal during the effectivepicture period of the next field. In this way, two types of storage timesignals (1/1000 sec. and 1/60 sec.) are alternately output.

In FIG. 2, a well known AE control circuit is shown at 20 which isadapted to receive a signal (for example, an image signal) from thecamera signal processing circuit 104 and output the control signal forexposure control. Also shown in FIG. 2 are a well known AF controlcircuit 22 adapted to output the control signal for focusing control, a1/2 frequency divider circuit 24 adapted to divide the frequency of thevertical blanking signal V_(BLK) by two, sample-and-hold circuits 26 and27, an inverter 28 and manual switches 29 and 30 for selecting betweenthe two alternatives, i.e., whether the sampling timing is to bedetermined by the output of the 1/2 frequency divider circuit 24 or bythe signal reversed at the inverter 28. The output of thesample-and-hold circuits 26 and 27 are respectively applied to thediaphragm drive circuit 106 and the focus drive circuit 107 to executeautomatic exposure control and automatic focus adjustment.

As discussed above, in accordance with this invention, the signalsindicative of a plurality of pictures of different exposures can beobtained in one field, so that the camera signal processing is partlychanged. The AE processing will be explained with reference to FIG. 2. Asignal derived from the image sensing element 103 is transmitted fromthe camera signal processing circuit 104 to the AE control circuit 20for AE control, and the AE control circuit 20 supplies a control signalto the diaphragm drive circuit 106 so that the AE control servo loop mayoperate. This control signal varies in such a manner that the output ofthe image sensing element 103 may be kept within an appropriate dynamicrange, i.e. the diaphragm 102 may be narrowed when light and widenedwhen dark.

When, however, the quantity of light changes, for example, with a periodof each field, a proper AE operation is not to be expected. For, theresponse of the AE servo takes a rather long time as compared with thechange in the period of the field or the frame. Therefore, the controlsignal for the AE operation is to be restricted in such a manner thatonly one of the two pictures of different exposures which are obtainedalternately may be employed. In FIG. 2, the sample-and-hold circuit 26is provided and the sampling timing is determined by the switch 29 insuch a manner that only such a control signal as corresponds to one ofthe two pictures (for example, that of 1/60 storage time) may beselected.

In FIG. 2, the sampling pulse is formed by the vertical blanking signal(V_(BLK)), and a signal which is reversed for each V_(BLK) signal isformed by the 1/2 frequency divider circuit 24. The signal thus formedis connected to the contact a of the switch 29, and the signal obtainedby reversing the output of the frequency divider circuit 24 at theinverter 28 is connected to the contact b of the switch 29. Accordingly,by selecting the contact a of the switch 29, the sampling pulse forusing the 1/1000 sec. storage signal of an odd field as the referencefor the AE loop control is applied to the sample-and-hold circuit 26.When, on the other hand, the contact b is selected, the sampling pulsefor using the 1/60 sec. storage signal of an even field as the referencefor the AE loop control is applied to the sample-and-hold circuit 26.The same may be said of the AF control circuit 22 for the AF control,the sample-and-hold circuit 27 and the switch 30.

In this embodiment of the present invention, a combination of 1/1000sec. storage time and 1/60 sec. storage time and the quantity of lightchanges by about four steps (2⁴ times), so that in the case of a camerausing a CCD image sensing element, an exposure adjustment to the subjectusing the 1/60 sec. storage time in an even field as the reference willlead to tendency to generate background white blankings in that evenfield, and in the case of an odd field with a light quantity diminishedby four steps, subject blackenings will often be generated. In thisexample, a case is assumed where the exposure is adapted to thebackground under a back light condition, and the setting may naturallybe made at a storage time other than 1/1000 sec., according to thesituation.

This invention utilizes such white blankings and/or blackenings in oneof the two types of fields in order to improve the picture quality. Inother words, parts generating white blankings or blackenings arereplaced with the corresponding parts in the other field (containing noblackening or white blanking because of the difference in exposure), thesignals of both fields being combined to compose a final image signal.The basic idea in this regard will be explained with reference to FIG.4(a). In FIG. 4(a), the main subject is represented by an uprightrectangle. A through (T) picture here means a real-time output from theimage sensing element 103, and a memory (M) picture or memory outputmeans the signal of an immediately foregoing field stored temporarily inthe image memory 204. The through pictures show the main subject asblackened for each odd field and as blanked for each even field.Further, since the memory pictures are composed of signals delayed byone field period, they show white blankings and blackenings in differentfields from those of the through pictures.

Consequently, an appropriate combination of the through pictures and thememory pictures will provide a satisfactory image which is free fromboth white blankings and blackenings. In other words, the signals of thethrough and memory pictures are compared with a predetermined threshold;they are arbitrarily assigned the value "1" when they are greater thanthis threshold, and the value "0" when smaller, thus enabling adesignation of white blankings and blackenings for each picture element.FIGS. 6(a) and 6(b) show the relation between the threshold and theluminance values of picture elements and their fields. The abscissa inFIG. 6(a) denotes the luminance levels and the ordinate denotes thefrequency of appearance of the luminance levels in a picture. As shownin FIG. 6(a), the threshold Thl is so set that luminance levelsexceeding it define blackenings, and the threshold Th2 is so set thatluminance levels exceeding it define white blankings. In other words,luminance levels lower than the threshold Th1 are judged to beblackenings and those higher than the threshold Th2 are judged to bewhite blankings. FIG. 6(b) shows the relation between the fields and thethresholds. As explained above, the white blankings and blackeningsalternate in the odd and even fields, so that the thresholds for theirdesignation also alternate with each field.

In this ways it can be judged which of the picture elements in eachfield shows a blackening or white blanking, and the result of thejudgment can be utilized to select a picture element of a properexposure from between the through and the memory pictures. For example,by obtaining the logical product of the judgment A and the judgment B,i.e., the product of the assigned values of the through and memorypicture, and selecting, in the case of odd fields, the through picturesignal for a picture element the logical product of which is "1", andthe memory picture signal for a picture element the logical product ofwhich is "0", and vice versa, in the case of even fields, the selectionflags as shown in FIG. 4(a) are obtained. The pictures shown in thelowest row in FIG. 4(a) correspond to what has been composed based onthe selection flags. As shown in this figure, a case was assumed inwhich the main subject underwent a uniform motion to check the influenceof the time axis deviation on the image; such a deviation proved to benegligible in practice.

FIG. 5(a) is a block diagram showing in detail that circuit part of thearithmetic circuit 202 in the processing section 200 in which thecomparison of the above thresholds Th1 and Th2 with the picture signalsand the formation of selection flags are effected. A Th switchingcontrol signal is a signal the "H" and "L" of which are reversed foreach field, as in the case of the FI signal, and transmitted through athreshold generating circuit 53 and an inverter 51, to be applied to asecond threshold generating circuit 52. The threshold generatingcircuits 52 and 53 generate, in accordance with their switching signals,the threshold Th1 or Th2 in the relation shown in FIG. 6(b). Comparisoncircuits 54 and 55 compare the memory and through pictures with thethresholds from the threshold generating circuits 52 and 53 and issuesignals A and B as the judgment results. An AND gate 56 takes thelogical product of the signals A and B and outputs a selection flagsignal. The switch 57 changes its position in accordance with said flagsignal to perform the selection between the memory and the throughpicture signals.

FIG. 5(b) is a block diagram showing another example of theconfiguration of that circuit part of the arithmetic circuit 202 in theprocessing section 200 in which the signal comparison with the abovethresholds Th1 and Th2 and the selection flag formation are carried out.Switches 501 and 502 have their positions changed by a switching controlsignal the "H" and "L" of which are reversed for each field (the FIsignal, for example). In the example shown, the switch 501 is connectedwith the contact b during an "H" period, the switch 502 then beingconnected with the contact M. During an "L" period, the switch 501 isconnected with the contact a, whereas the switch 502 is controlled alsoby the output of an AND circuit 507. Two threshold generating circuits503 and 504 generate the thresholds Th1 and Th2, respectively, which areshown in FIG. 6(a). A comparison circuit 505 compares the output signalfrom the memory 204 (memory out) with the threshold Th1 to output thejudgment B, whereas a comparison circuit 506 compares the output signalof the A/D converter 201 (through picture) with the threshold Th2 tooutput the judgment A. The AND circuit 507 takes the logical product ofthe comparison circuits 505 and 506 and outputs it as a selection flag.As explained above, the switch 502 is controlled by the output of theAND circuit 507.

FIG. 4(b) shows the operation of a circuit which is of the configurationshown in FIG. 5(b). In this figure, "memory in" means a picture which iswritten to a field memory (the memory 204 in FIG. 5(b), and the "memoryout" means a picture which is read out of the above memory. During anodd field period, the through picture (T) is directly written to thememory 204, whereas during an even field period, the through picture (T)and the odd field picture which is one field before it, which is amemory out, are combined after a predetermined weighing and written tothe memory 204.

When composing signals of different storage times shown as A and B inthe storage time column in FIG. 3, the composition is effected bycombining two pictures such as A1 and B1, A2 and B2, and a picturecomposed of such a combination as B1 and A2 is avoided. By doing so, thestorage periods for obtaining two pictures to be combined become closeto each other, so that a composed picture with few blurs can beobtained.

Further, during an even field period, the threshold Th1 is compared withthe memory out, and the threshold Th2 with the through picture; when theinput data are greater than the thresholds, the value assigned is "1",and when smaller, the value assigned is "0". Then they are output as thejudgments A and B for the above through picture (T) and the memory out,respectively.

In accordance with these picture element judgments, signals are selectedfrom the image signals shown as part of the through picture (T) and thememory out for each picture element area. The result of the selection isrepresented by the even field period pictures shown in the lowest columnin FIG. 4(b). During an odd field period, an output image (memory out)which is one field period before it is output again. By thus performingpicture composition using pictures which are close to each other inrespect of time, image deterioration in border areas can be avoided evenfor moving objects.

A picture with clear differences in luminance may be divided into twoareas, i.e., the subject and the background. In such a case, it isdesirable that in each of the areas, only the signals of one type ofstorage time be used, avoiding a change in storage time within a singlearea. This is partly because a proper tonal expression would then behindered in that area and partly because the border portions between thedifferent storage times in each area flicker, resulting in a picturethat may be irritating to watch. Accordingly, in order to avoideffecting such a change frequently, it is desirable that a threshold tobe used as a reference for the change be set, taking into considerationthe information on the peripheral picture elements. In view of this, thethresholds Th1 and Th2 may be provided with hysteresis characteristics,as shown in FIG. 6(c). In other words, the threshold generating circuits52 and 53 shown in FIG. 5(a), for example, may be so designed that theygenerate, in accordance with their switching signals, the thresholdsTh1, Th1', or Th2, Th2', in the relation shown in FIG. 6(c). The abovehysteresis characteristics may be such that when picture elementinformation with a value below the Th1 in FIG. 6(c) is received, thethreshold Th1 is raised to Th' (Th' being greater than Th1) and theoutput of the comparison circuits 54 and 55 is prevented as far aspossible from being reversed. When, however, the output of thecomparison circuits 54 and 55 has been reversed, the threshold Th1' isrestored to Th1. The same can be said of Th2; when information with avalue above Th2 is received, the threshold Th2 is lowered to Th2' (Th2'being less than Th2). By doing this, the distinction between the subjectand the background can be made more correct, the isolated points in thepicture resulting from the judgment being reduced, which makes itpossible to effect a more natural picture composition. Also, the borderportions between the subject and background become less subject toflickering, whereby a more natural image can be obtained.

FIGS. 7(a) to 7(d) are tonal characteristic drawings. The full line inFIG. 7(a) represents the characteristics of an ordinary video camera; upto 100%, the input and output are shown as linear. For an input beyondthat value (100 to 400%), the characteristics are illustrated with asmoothly inclined line; the relation is called a knee effect. Supposethe point where such a change begins is P1; the point will then move tothe position P2 when the shutter speed is high. Here, it is to beassumed that P1 represents an exposure of 1/60 sec. and P2 that of 1/250sec. obtained by a two step exposure change. When there is a differenceof four steps, as in the case described above, the relation may berepresented by the lines (1) and (5) in FIG. 7(d). In this figure, theline (1) represents 1/60 sec., the line (2) 1/125 sec., the line (3)1/250 sec., the line (4) 1/500 sec. and the line (5) 1/1000 sec. Inaccordance with this invention, a characteristic with a desired curvecan be composed out of two characteristics with different inclinations.Examples of such a composition is shown in FIGS. 7(b) and 7(c).

The method of composing tonal characteristics will now be explained indetail. By way of example, a case will be taken where it is socontrolled that either the point P1 representing the change from thelinear to the knee effect section (when the shutter speed is low) or thepoint P2 (when the shutter speed is high) reaches the 100% point.Thresholds are provided as shown in FIG. 6(a) so as to perform judgmentover white blankings and blackenings on pictures, the tonalcharacteristics varying with the setting of the thresholds. Thevariation is shown in FIG. 7(b), in which the characteristics (1) to (3)show how the position where the high shutter side is selected inswitching picture elements with the switch 57 moves to the highluminance side when the threshold Th2 for judging white blankings isgradually raised.

While in FIG. 5(a) one of the two types of picture element signals isselected with the switch 57, it is also possible to obtain the desiredsignal by processing the data on the corresponding two picture elements.In connection with this method, FIG. 7(c) shows, by way of example, thechange in characteristics when different operation methods are selected,with the characteristic (2) in FIG. 7(b) being used as the reference.The characteristic (2) in FIG. 7(c) is identical with the characteristic(2) in FIG. 7(b). The characteristic (1) in FIG. 7(c) represents thecase where mean value processing is performed using the picture elementdata of a picture which has been judged to contain white blankings andthe corresponding picture element data of another picture. Thecharacteristic (3) represents the case in which subtraction processingis performed. Suppose, for example, the data on two correspondingpicture elements judged to contain white blankings are D1 and D2, andthe processing result is D, D=(D1+D2)/2 with mean value processing andD=D1-k (D1-D2) with subtraction processing, where k varies in accordancewith the threshold set. In the case of the characteristic (3) in FIG.7(c), k is about 1.88. Apart from the mean value processing andsubtraction processing, what is called off-set processings such asconstant addition and constant subtraction may be utilized, or they maybe used jointly.

Further, when the subject is properly exposed, the condition of thebackground image (such as the degree of white blanking) becomesadjustable, so that a more natural representation and an elaboratedesign can be realized. Naturally, it is also possible to adjust thedegree of blackening of the subject, with the background properlyexposed. The adjustment range of such image conditions can be utilizedwithin an infinite characteristic range by the picture element operationmethod as well as by the combination of shutter speeds.

In the above embodiment, the time resolution is substantially 30pictures per second, which is about the same as a frame storage CCDimage sensing element or the like. An embodiment will now be explained,according to which two pictures are taken in a single field, so that atime resolution equal to that of a field storage CCD image sensingelement may be realized. The modified sections in the example are shownin FIG. 8 and the timing chart thereof in FIGS. 9(a) to 9(d). In thisembodiment, the signal of the image sensing element 103 is read out at aspeed higher than that of an ordinary video rate; the signal thenundergoes a time axis transduction and is restored to the normal rate.Field memories 90 and 91 are each provided with a memory storagecapacity corresponding to the picture information for a single field; inthe memory 90, the 1/1000 sec. storage signal is delayed so as to besynchronized with the 1/120 sec. read-out timing, and in the memory 91,the length of the time axis is doubled so as to change the 1/120 sec.picture signal into a 1/60 sec. NTSC signal. FIGS. 9(a) to 9(d)correspond to the signals (a) to (d) in FIG. 8. The switch 57 forms thecomposition output (c) by switching the 1/120 sec. storage signal andthe 1/1000 sec. storage signal in accordance with the output of acontrol circuit 92. This arrangement enables a resolution for each fieldto be obtained, as shown in FIG. 9(d).

FIG. 10 shows in detail another example of the control circuit 108 inFIG. 1. A master clock generator 40 generates master clock signals forthe interior of the control circuit 108 in accordance with the referencesignal from outside. A clock generator 41 for 1/1000 shutter generatesclock signals for high speeds in accordance with the above master clocksignals, and a clock generator 42 for 1/60 shutter generates clocksignals for low speeds in accordance with the above master clocksignals. A switch 45 changes its position for each field and applies theoutput of the clock generators 41 and 42 alternately to the drivecircuit 105. An AE control signal generator 43 generates AE controlsignals for diaphragm control, based on the image signals from thecamera signal processing circuit 104. A control signal retaining circuit44 retains the AE control signals for the period of one field. A switch46 changes its position for each field and applies the output of the AEcontrol signal generator 43 and the control signal retaining circuit 44alternately to the control circuit 106. A switching signal generator 47controls the switching operations of the switches 45 and 46 for eachfield. The switches 45 and 46 change their positions synchronously.

In this embodiment, a clock generator is provided for low and highspeeds, and the clock signals thereof are changed in accordance with theoutput signals of a switching signal generator generating signals foreach field, so that the circuit configuration and operation aresimplified, which is suited to moving pictures.

While in the above embodiments pictures of different exposures areproduced by varying the storage time of the image sensing element, it isalso possible to attain the production of pictures with differentexposures by varying exposure controls such as a high-speed diaphragm orshutter at a high speed. Further, this can also be realized byelectrically controlling a dimmer filter, as in the case of PLZT.

As will be appreciated from the foregoing description, the presentinvention permits the dynamic range to be substantially enlarged, sothat even under a back light condition, a properly exposed image in theform of a moving picture can be obtained not only for the subject butalso for the background.

In the above embodiments, the operation loop of the camera for movingpictures is set in such a manner that the response is of several secondsdurations, thus avoiding an abrupt change of pictures. When, forexample, the exposure times of 1/1000 sec. and 1/60 sec. are alternatedfor each field, the same response is shown in the AE loop in FIG. 2 aswhen the exposure time of 1/250 sec. is adopted throughout. This can beregarded as a stationary error, which can be overcome by adding anoffset bias corresponding to the exposure difference, to the AE loop.FIG. 11 shows an example of such a modification of the circuit shown inFIG. 2. In this example, a bias generating circuit 80 generates suchbiases as discussed above; an adder 81 adds the bias to the AE loop; andan adder 82 adds a similar offset bias to the AF loop. The same can besaid of AWB (automatic white balance adjustment). This arrangement makesit possible to stabilize the AE, AF or AWB with a constant servo gain.

FIG. 12 shows a third embodiment of this invention and FIGS. 13(a) to13(i) show the operational timing thereof. In these figures, what arefurnished with the same reference numbers as those in FIGS. 1 to 11denote the same components. The output of the camera signal processingcircuit 104 include the luminance (Y) signal (see FIG. 13(a)), whichundergoes sampling at the A/D converter 109 with 4f_(sc) (f_(sc) denotesthe color subcarrier frequency), and quantization into 8-bit signals andis then supplied to the contact b of a switch 112. To the contact a ofthe switch 112 is supplied a composed picture, which is the output of aswitch 116. The switch 116 changes its position for each field with theFI signal shown in FIG. 3 (or a signal equivalent to this) and connectsthe contact a for an even field and the contact b for an odd field. Theoutput of the switch 112 is supplied to the contact a of the switch 116by the memory 113, delayed by one field period. To the contact b of theswitch 116 is supplied the output of an A/D converter 109. The positionof the switch 116 is changed by the selection flag (see FIG. 4(a))output from a control circuit 120. The output of the switch 116 issupplied to the contact a of the switch 112, and at the same time, it isconverted into an analog signal by a D/A converter 118 and supplied to amixing circuit 125, where it is mixed with a color-difference signal tobe recorded in a video tape recorder 400.

The output of the camera signal processing circuit 104 also includes twocolor-difference signals (R-Y) and (B-Y) (FIG. 13(c) and FIG. 13(d)),which are switched by a switch 110, with the timing shown in FIG. 13(e).FIG. 13(f) shows the output of the switch 110, which undergoes samplingat an A/D converter 111, with 2f_(sc) pulses of the timing shown in FIG.13(g), and quantization into 8-bit signals, and is then supplied to thecontact b of a switch 114. The processing at the switch 114, the fieldmemory 115, the switch 117 and the D/A converter 119 is the same as forthe Y-signal, except that the memory capacity of the field memory 115may be half that of the field memory 113. The output signal of the D/Aconverter 119 (FIG. 13(f)) is sampled and synchronized bysample-and-hold (S/H) circuits 121 and 122, with the timing shown inFIG. 13(h) and FIG. 13(i). After this sampling and holding, the R-Ysignal (FIG. 13(c) and the B-Y signal (FIG. 13(d)), which have beenrestored to their original state with a low pass filter (not shown),effect quadrature two-phase modulation of the f_(sc) in a modulationcircuit 124. The output of the modulation circuit 124 is superimposed onthe Y-signal and supplied to the video tape recorder 400, as an NTSCsignal.

FIG. 14 shows that portion of the control circuit 120 where selectionflags are generated. Threshold generating circuits 503 and 504 generatethe thresholds Th1 and Th2 shown in FIG. 6(a), respectively. Acomparison circuit 505 compares the output signal (memory out) of thefield memory 113 with the threshold Th1 to output the judgment A, and acomparison circuit 506 compares the output signal (through picture) ofthe A/D converter 109 with the threshold Th2 to output the judgment B.An AND circuit 507 takes the logical product of the judgments A and B ofthe comparison circuits 505 and 506 and applies it to one input of an ORcircuit 509. To the other input of the OR circuit 509 is applied the FIsignal, so that the output of the OR circuit 509 is "H" in an ODD field.

The output of the OR circuit 509 is sampled and held by an S/H circuit510 with the timing shown in FIG. 13(h) or FIG. 13(i). This is to beeffected because of the following circumstances: the sampling frequencyis different for the luminance and the color signals, so that when aselection flag is formed with all the luminance signals selected, thereoccurs the inconvenience that although the output of the A/D converter109 is selected for the R-Y signal of a sampling point on the picture,the output of the field memory 113 is selected for the B-Y signal inthat position. In other words, this inconvenience consists in the factthat the R-Y signal is obtained from one of two pictures of differentstorage times, and the B-Y signal from the other; as a result, the colorsignal obtained from the modulation circuit 124 does not correspond tothe actual color of the subject. To overcome this problem, thisembodiment provides an S/H circuit 510 shown in FIG. 14, with itssampling rate f_(sc). The output of the S/H circuit 510 is applied tothe switches 116 and 117, which are placed at the contact a for "H", andat the contact b for "L".

By the above arrangement, this embodiment makes it possible to avoid theerror of performing the composition by obtaining the R-Y signal from oneof two pictures of different storage times and the B-Y signal from theother.

In the above embodiment, one of the Y-judgment signals (corresponding tothe output of the AND circuit 507) is used as the representative valueby means of the circuit explained in FIG. 14 in order to produce asingle C-selection signal (corresponding to the output of the S/Hcircuit 510) from among predetermined Y-judgment signals. Apart fromthis method, the C-selection signal can be produced by majorityprocessing in which the C-selection is effected in accordance with thenumerical judgment result, or mean value processing in which the meanvalue of a predetermined Y-signal section is regarded as the C-judgmentresult.

The above embodiment of this invention makes it possible to conductcomponent color signal processing, so that the basic picture quality canbe improved and pictures of a generally higher quality can bephotographed.

A fourth embodiment of this invention will now be explained withreference to FIGS. 15 to 21. In FIG. 15 are shown a subject 710, aphotographic lens 712, a diaphragm 714 and an image sensing element 716.The incident light from the subject 710 entering through the lens 712 isrestricted in intensity by the diaphragm 714 and applied to thephotoelectric conversion surface of the image sensing element 716. Acamera signal processing circuit 718 performs various signal processingoperations including γ correction in the same way as an ordinary videocamera. A diaphragm control circuit 720 controls the diaphragm 714, anda drive circuit 722 drives the image sensing element 716. A gate circuit724 allows part of the output of the camera signal processing circuit718 to pass, and a speed switching circuit 726 changes the chargestorage time of the image sensing element 716.

The output of the camera signal processing circuit 718 is divided by adecoder 728 into a Y-signal and a color-difference signal (C) of a timedivision multiple base band, which are applied to signal processingcircuits 732Y and 732C through A/D converters 730Y and 730C,respectively. In the signal processing circuits 732Y and 732C, pictureelement data are transduced, as described later, and applied to anoutput processing circuit 736 through D/A converters 734Y and 734C. Theoutput processing circuit 736 forms a composite video signal from theinput signal and outputs this signal.

A judgment circuit 738 judges whether there are blackenings or whiteblankings from the Y-signal, and in accordance with the judgment result,provides a switching command signal for the speed switching circuit 726,a control signal for the gate circuit 724, etc.

The processing circuits 732Y and 732C will now be described in detail.While only the processing circuit 732Y is shown in FIG. 16, theprocessing circuit 732C may substantially be of the same configuration.In FIG. 16, the output of the A/D converter 730Y is applied to switches750 and 758. The switch 750 is always placed at the contact b in thenormal mode (i.e. when there are no white blankings or blackenings onthe photographed image), and is alternately placed at the contacts a andb, with 1/60 sec. as a unit, in the picture element transduction mode(i.e. when there are some white blankings or blackenings on thephotographed image). In the picture element transduction mode, the drivecircuit 722 drives the image sensing element 716 in such a manner thatit alternates the 1/60 sec. charge storage and the 1/1000 sec. chargestorage for each field, and the switch 750 is placed at the contact afor a 1/1000 sec. storage picture and at the contact b for a 1/60 sec.storage picture. The 1/60 sec. storage picture at the contact b of theswitch 750 is supplied to the judgment circuit 738 and a multiplexer756.

In the normal mode, the multiplexer 756 outputs 1/60 sec. storagepictures supplied for each field, and in the picture elementtransduction mode, it performs, when in an odd field, picture elementtransduction between a 1/60 sec. storage picture from the contact b ofthe switch 750 and a 1/1000 sec. storage picture from the field memory754 to output the picture element transduction picture; when in an evenfield, it outputs the image from the field memory 754 which has alreadyundergone picture element transduction, as it is. The switch 758, whichis normally placed at the contact a, is placed at the contact b duringthe vertical blanking period. This arrangement is made for the purposeof forming interlace signals.

The operation in the picture element transduction mode of the switch752, the field memory 754 and the multiplexer 756 will now be describedin more detail referring to FIGS. 17(a) to 17(g). FIG. 17(a) representsthe switching control signal for the switches 750 and 752, which areplaced at the contact a for "H" and at the contact b for "L". FIG. 17(b)represents the switching control signal for the switch 758, which isplaced at the contact a for "H", and at the contact b for "L". FIG.17(c) shows the charge storage operation of the image sensing element716, the a_(n) and b_(n) being representative of the charge storagetimes (1/1000 sec. and 1/60 sec. in this example). FIG. 17(d) shows theoutput of the image sensing element 716. FIG. 17(e) shows the signalwritten to the field memory 754, FIG. 17(f) shows the signal which isread out of the field memory 754, and FIG. 17(g) shows the output of themultiplexer 756. The field memory 754 is a memory of the type which iscapable of performing both writing and reading operations.

The a₁, which is a 1/1000 sec. storage picture, is delayed by one fieldperiod by the field memory 754, picture element transduction beingperformed by the multiplexer 756, between the a₁ and the 1/60 sec.storage picture b₁ of the next field. The image MIX-1 obtained throughthe picture element transduction is supplied to the switch 758, and atthe same time, written to the field memory 754 through the intermediaryof the switch 752. The image MIX-1 is applied to the multiplexer 756,delayed by one field period. At this time, only the image MIX-1 isapplied to the multiplexer 756, so that the multiplexer 756 outputs theimage MIX-1 to the switch 758 again.

After that, similar processings are carried out, the multiplexer 756outputting MIX-1, MIX-1, MIX-2, MIX-2, . . . Here, it may be thought tobe possible to improve the time resolution by performing picture elementtransduction between b₁ and a₂ in the next field, subsequent to thetransduction between a₁ and b₁. However, the discrepancy in time betweenb₁ and a₂ is too great, so that it may as likely as not give rise toinconveniences in the case of moving pictures.

FIG. 18 is a block diagram showing a specific example of the judgmentcircuit 738. The Y-signal of 1/60 sec. storage from the processingcircuit 732Y (more specifically, from the contact b of the switch 750)is applied to a mean value calculation circuit 760 and a comparisoncircuit 762 adapted to detect white blankings. The mean valuecalculation circuit 760 divides a picture into a plurality of areas, asshown, for example, in FIGS. 20 and 21, and calculates the mean value ofthe luminance levels in the areas and supplies the result to anarithmetic circuit 764. The arithmetic circuit 764 calculates thedifference between the maximum and minimum values of the mean values ofthe areas and supplies control signals to the speed switching circuit726 in such a manner that the picture element transduction mode isselected when the calculated value is larger than a predetermined valueand that the normal mode is selected when the calculated value is equalto or smaller than the predetermined value. The arithmetic circuit 764consists of an ordinary microcomputer. Further, the mean luminancelevels of the areas calculated by the mean value calculation circuit 760can also be utilized for the purpose of determining the photometricframe for diaphragm adjustment, the arithmetic circuit 764 outputtingthe photometric frame information to a window generating circuit 766.

The comparison circuit 762 compares the Y-signal from the processingcircuit 732Y with the threshold from a threshold switch 768, and outputs"L" when the signal is below the threshold and "H" when it is above thethreshold. In the initial stage, the threshold switch 768 selects thethreshold Th2 shown in FIG. 19; when the output of the comparisoncircuit 762 becomes "H", it changes the selection threshold to Th1, andwhen the output of the comparison circuit 762 becomes "L", it selectsthe threshold Th2 again. By providing such hysteresis characteristics,frequent issuance of the white blanking judgment at isolated points suchas noises can be avoided. The output of the comparison circuit 762 issupplied to the processing circuit 732Y as the Y-control signal and tothe processing circuit 732C as the C-control signal. The processingcircuits 732Y and 732C select a 1/60 sec. storage picture when theoutput of the comparison circuit 762 is "L", and a 1/1000 sec. storagepicture when the output is "H".

While it may be possible to adopt in the judgment circuit 738 anarrangement in which a 1/1000 sec. storage picture is used as the inputluminance signal in order to judge over blackenings, the aboveembodiment employs the arrangement in which a 1/60 sec. storage pictureis input to the judgment circuit 738 because white blankings are moreeasily detected than blackenings and the switching between the normalmode and the picture element transduction mode must be carried out underthe same storage conditions.

As will be appreciated from the above description, this embodiment makesit possible to photograph not only the subject but also the backgroundwith a proper exposure even under a back light condition, with thedynamic range being substantially widened. Further, by supervising theluminance level during the normal photographing and by performing thenormal photographing when the exposure is proper, deterioration in thetime resolution can be avoided as long as the exposure is appropriate.

What is claimed is:
 1. An image sensing apparatus comprising:(a) imagesensing means for sensing an image; (b) image composing means forcomposing an image with a plurality of image signals output from saidimage sensing means; (c) comparison means for comparing a luminancesignal level of said plurality of image signals with a predeterminedreference level, said comparison means evaluating the luminance signallevel distribution; (d) controlling means for controlling a composingcondition based on a comparison result of said comparison means.
 2. Animage sensing apparatus according to claim 1, wherein said imagecomposing means include a selection circuit.
 3. An image sensingapparatus according to claim 1, wherein said comparison means evaluate ahistogram of luminance signal level distribution.
 4. An image sensingapparatus according to claim 1, wherein said control means change aborder of said plurality of image signals.
 5. An image sensing apparatuscomprising:(a) image sensing means for sensing an image; (b) imagecomposing means for composing an image with a plurality of image signalsoutput from said image sensing means; (c) histogram evaluate means forevaluating a histogram of a luminance signal level distribution of saidplurality of image signals; (d) controlling means for controlling acomposing condition based on an output of said histogram evaluate means.6. An image sensing apparatus according to claim 5, wherein said imagecomposing means include a selection circuit.
 7. An image sensingapparatus according to claim 5, wherein said control means change aborder of said plurality of image signals.